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The determination of the error probability of a data transmission system 
in the presence of intersymbol interference and additive gaussian noise is 
a major goal in the analysis of such systems. The exhaustive method for 
finding the error probability calculates all the possible states of the received 
signal using an N sample approximation of the true channel impulse 
response. This method is too time-consuming because the computation 
involved grows exponentially with N. The ivorst-case sequence bound 
avoids the lengthy computation problem but is generally too loose. 

In this paper, we have developed a new method* which yields the error 
probability in terms of the first 2k moments of the intersymbol interference. 
A recurrence relation for the moments is derived. Therefore, a good approxi- 
mation to the error probability of the true channel can be obtained by 
choosing N large enough, and the amount of compulation involved increases 
only linearly with N. The series expansion is shown to be absolutely 
convergent, and an upper bound on the series truncation error is given. 
In order to show the improvement provided in this new method, it is com- 
pared with the Chernoff bound technique in three representative cases. 
An order of magnitude improvement in accuracy is obtained. 

I. INTRODUCTION 

An important problem in the analysis of binary digital data sys- 
tems is the determination of the system performance in the presence 
of intersymbol interference and additive gaussian noise. Since it is 
usually the most meaningful criterion in designing a digital data 



*In April 1970, the authors were advised by R. W. Pulleyblank that a similar 
method was discovered independently by M. Celebilcr and 0. Shimbo to be 
presented in a paper which will be published in Conference Record, ICC, 1970. 
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system, the error probability is chosen as the measure of the system 
performance. 

Two alternatives are available at present. The first alternative 1 ' 2 
considers a truncated JV-pulse-train approximation of the true channel. 
The error probability is calculated by evaluating the conditional error 
probability of each of 2 N possible data sequences and averaging over 
all 2 N sequences. Since each calculation of the conditional error prob- 
ability takes a great deal of computer time, the number of sequences 
must be held to several thousand. 3 This limitation leads to a poor 
approximation of the true channel, and the error probability so ob- 
tained is not very useful. The second alternative evaluates an upper 
bound of the error probability by either the worst-case sequence 3 or 
the Chernoff inequality. 4,5 In many cases, the bound is too loose. 

In this study we have developed a new way to evaluate the error 
probability in terms of the first 2/c moments of the intersymbol inter- 
ference. It provides a significant improvement in accuracy over the 
worst-case sequence bound or the Chernoff bound. The computations 
increase only linearly with N. Thus a good approximation of the true 
channel may be obtained. The convergence of this alternative is proved. 
Throughout, additive gaussian noise and independence of information 
digits are assumed. The generalization to a multilevel system is 
straightforward; hence, only binary systems will be considered in 
this study. 

II. BRIEF DESCRIPTION OF THE SYSTEM 

A simplified block diagram of a binary amplitude modulation (AM) 
data system is shown in Fig. 1. We assume that a single s(t) having 
amplitude a ( is transmitted through the channel every T seconds. The 
system transfer function is 

R(u) = S(a>)T(a)E(u) (1) 

where s(t) and r(t) are the Fourier transform pair of S(u>) and R(co), 
respectively. In the absence of channel noise, a sequence of input 
channel signals 

CO 

£ a ( s(t ~ IT), (2) 

will generate a corresponding output sequence 

CO 

£ a<r(t - IT), (3) 

l = — 00 

where \a ( ] is a sequence of independent binary random variables, 
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Fig. 1 — Simplified block diagram of a binary AM data system. 

a t = ±1, and satisfies 

P T (a ( = 1) = P r (a< = -I) = h 



= — 00 



, -1,0,1, 



(4) 



We also assume that additive gaussian noise is present in the system. 
Thus the corrupted received sequence at the input to the receiver 
detector is 



y(t) = E a f r(t - IT) + nfl), 



(5) 



where n(t) is additive gaussian noise with a one-sided power spectral 
density of a 2 watts/cps. 

At the detector, y(t) is sampled every T seconds to determine the 
transmitted signal. At sampling instant t , the sampled signal is 



y(t ) = a r(t ) + E «<K«o - IT) + n(t ). 



(6) 



l = -x 

1*0 



The first term is the desired signal while the second and the third 
terms represent the intersymbol interference and gaussian noise re- 
spectively. 

It is well known that the optimum (minimum error probability) 
decision level is zero. Thus the error probability is given by 



P, = Pr 



E a,r(t - IT) + n{t ) ^ 



_f=-» 

1*0 



)j ^r(t )i- 



(7) 



For the real system we are interested in, we may assume that the 
a t r(t — tT)'s are uniformly bounded and E<*o a c r(t — IT) converges 
absolutely.* For example, in a system having an open binary eye, 



* Finite truncated pulse-train approximation will be used for those pulses with 
absolutely divergent intersymbol interference. 
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X^o | r(t — tT) I is less than r(t ). Thus by Kolmogorov's Three-Series 
criterion, 6 it can be easily shown that 2^*0 a t r( J-o ~ tT) converges 
absolutely to a random variable. 

Equation (7) can be calculated by evaluating the expected value of the 
conditional expectation of the error probability for a given random 
variable Xl^o Q>ir{t — IT); therefore, 

P. = I —?=- f exp [- \y - r(t ) - X\ 2 /2a 2 \ dy dF(x), (8) 

■"nil X V2x <T J -<*> 

where F(X) is the distribution function of the random variable X, and 
A' = E^oMfc -IT). 

HI. SERIES EXPANSION OF P e 

With the exception of a few special cases, equation (8) is generally 
difficult to solve. The existing solutions are either too time-consum- 
ing 1 ' 2 or inaccurate. 3 ' 4,6 

We have found that equation (8) can be evaluated in terms of an 
absolutely convergent series involving moments of the intersymbol 
interference. Furthermore, the moments can be obtained readily 
through recurrence relations. Therefore, the computation time is sig- 
nificantly reduced in comparison with the exhaustive method. 1,2 The 
absolute convergence and the recurrence relations for the moments are 
given in Appendix A and B respectively. 

Expanding equation (8), we obtain the following expression for the 
error probability, 

P. = *e rf c(-^J +? ^.(^M M .^ 



■"■-- il-^J ' - ' ' - 



-P..+ EP..., (9) 

*=1 
where H 2 k-i(x) is a Hermite polynominal, M- lk is the 2/cth moment of 
the random variable X, and 



2 r~ z 

erfc (—x) = — 7= / exp (— z) dz. 



(10) 



The first term in equation (9) represents the nominal system error 
probability due to additive gaussian noise alone while the summation 
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represents the degradation of the system performance due to intersym- 
bol interference in the additive gaussian noise environment. 

3.1 Convergence Property 

In Appendix A we have shown that equation (9) is an absolutely 
convergent series. Therefore, the error probability can be evaluated 
by taking a finite number of terms, 

P. = lP. It +Bii,, (11) 

k = 

where R 2 k represents the truncation error and is upper bounded by 
A (2K-3)!! -/ST-2 * -L. 

[exp-('^)]-[glK/o-^)r 




E I K«o - 00 



= t/« • (12)* 

Thus for a given truncation error bound, e, we may always find a 
positive integer, K, such that 

U 2K ^ e. (13) 

For a real system, the truncation error is generally much smaller 
than e. Therefore, fewer terms are needed in evaluating the error prob- 
ability. 

3.2 Evaluation of Moments 

The series expansion of equation (9) can be readily evaluated if we 
can determine the moment, M 2fc . The M 2k 's are given by 

M 2k - [ X 2k dF{X). (14) 

To evaluate M 2 ,. according to equation (14) requires the knowledge of 
dF{X) ; this is just as difficult to obtain as the evaluation of the error 
probability given by equation (8). However, we have found it possible 
to obtain a recurrence relation for M 2 u by examining the first deriva- 



* (2/v - 3) ! ! = (2/v -3) • (2K - 5) ■ ■ • 3-1. 
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tive of the characteristic function. The recurrence formula makes the 
series expansion approach feasible, and is derived in Appendix B: 

Mm = ~{t (2- Z j)(-l) , '^ 2 (*-.>/ (2, '- 1) (0)} , (15) 

where 

M - 1 (16) 

/(2 ,._ 1)(0) m 2 2 -(2 2 - - 1) ! ^ ! £ [r{tQ _ tT)r (17) 

and B 2 t's are Bernoulli numbers. 

3.3 Truncated Pulse-Train Approximation 

For any real binary system, the message must be time-limited to a 
finite number of symbol durations, or we may even assume that r{t) is 
time-limited to, say, N symbol durations. Thus the error probability 
may be calculated by evaluating the conditional error probability for 
each of 2 N possible data sequences and then averaging over all 2 N 
sequences. Since the number of possible data sequences grows exponen- 
tially with N, it would be impractical to evaluate the error probability 
by this straightforward method even with a digital computer. Hence, N 
must be confined to a small number; the error probability so obtained 
could at best be a poor approximation of the true error probability. 
However, in equation (9), the amount of computation involved grows 
only linearly with N. Therefore, the pulse train can be truncated at 
any desired point to assure a good approximation of the true channel. 

IV. APPLICATIONS 

The error probabilities for certain cases are calculated by equation 
(9) to determine the accuracy and the convergence of this new method. 

4.1 Casel: Data Set 203 7 

A 2400-baud DDD option of the Data Set 203 operating over a 
channel having symmetrical parabolic delay distortion, as shown in 
Fig. 2, is considered in this case. The group delays at the carrier and 
the lower 3-dB frequencies are 0.6 ms relative to the center of the 
signal spectrum. The channel we considered is worse than a worst- 
case-C2 line. A 5-tap mean-square equalizer is used by the receiver 
to equalize the channel. A truncated 34-pulse-train approximation (19 
samples after and 15 samples before the sampling instant t ) for the 
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Fig. 2 — Channel group-delay-frequency response. 

equalized output impulse response was used. The equalized binary 
eye is about 70 percent open in this case. The input signal-to-noise 
ratio is 14 dB. The error probabilities at the equalizer output evaluated 
by equation (9) and the Chernoff inequality are shown in Fig. 3. 
Curve (a) is the Chernoff bound. Curve (b) is the error probability 
evaluated by taking a finite number of terms in equation (9) . Curve 
(c) is the truncation error bound given by equation (12). It can be 
seen that taking the first nine terms in equation (9) assures less than 
one percent truncation error in evaluating the error probability. In 
this case, however, the actual series converges after only four terms. An 
improvement in accuracy by a factor of 15 is realized by this series 
expansion method compared to that obtained by Chernoff inequality. 

4.2 Case 2: Ideal Channel and Ideal Band-Limited Pulse 
The received pulse is assumed to have the form, 



r(t)i = 



sin wt/T 

Tt/T 



(18) 



The signal-to-noise ratio at the nominal sampling instant is taken 
to be 16 dB. In the absence of intersymbol interference, the system 
error probability is 10 -10 . For a truncated 11 -pulse-train approxima- 
tion, the exact error probabilities and the error probabilities evaluated 
by taking a finite number of terms in equation (9) for different values 
of sampling instant and number of terms and equation (12) are shown 
in Figs. 4-5. It can be seen from these figures that the series con- 
verges more rapidly for smaller values of the quantity 



q(Q = [Z I r(ta - IT) \/<r 
[e.g., in this case, g(0.05T) = 1.96]. 
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Fig. 3 — Comparison of error probabilities obtained by Chernoff bound and series 
expansion method. (S/N)i„ P „t = 14 dB; data set 203 (2400-Baud Option); 5-tap 
mean square equalizer; parabolic delay distortion channel (see Fig. 2). 

The series starts to oscillate when q(t ( >) is not small [e.g., q(0.2T) 
= 30.8] . At t„ = 0.2T, the series did not converge well for the first 
eight terms in equation (9). However, it will converge to the exact 
value eventually. The error probabilities obtained by Chernoff 
bound, 5 exact calculation, and equation (9) are shown in Fig. 6. It is 
clear that this new alternative provides a significant improvement over 
the Chernoff bound. 

4.3 Case 3: Ideal Channel and Fourth-Order Chebyshev Pulse 5 

In this case, a fourth-order Chebyshev filter is used. The received 
pulse is 
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r(t) = A, cos ( Wl \t\/T - *0-exp [-a, | Z |/T] 

4- A 2 cos (co 2 | i \/T - *,) -exp [-«> | t |/T], (19) 

with 

A x = 0.4023, A 2 = 0.7163, 

co x = 2.839, u. = 1.176, 

*, = 0.7553, $ 2 = 0.1602, 

a, = 0.4587, a 2 = 1.107. 

The signal-to-noise ratio at the nominal sampling instant is taken to be 
16 dB. For a truncated 11 -pulse-train approximation, the exact error 
probabilities and the error probabilities obtained by taking a finite 
number of terms in equation (9) for various sampling instants and 
numbers of terms are shown in Figs. 7-8. The error probabilities ob- 
tained by the Chernoff 5 bound, the exact calculation, and equation (9) 
are shown in Fig. 9. The same results as in case 2 are observed. 

V. SUMMARY AND CONCLUSIONS 

In this study we have developed a new method of evaluating the 
error probability for synchronous data systems in the presence of 
intersymbol interference and additive gaussian noise under the fol- 
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Fig. 4 — Error probabilities versus number of terms in equation (9). Ideal band- 
limited signal. 11-pulsc truncation approximation; sampling instant, t = 0.05 T; 
(S/N) = 16 dB. 
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Fig. 5 — Error probabilities versus number of terms in equation (9). Ideal band- 
limited signal. 11-pulse truncation approximation; sampling instant, t = 0.2 T; 
(S/N) = 16 dB. 

lowing assumptions. First, the information digits are identically and 
independently distributed. Second, the intersymbol interference con- 
verges absolutely. (For those pulses with absolutely divergent inter- 
symbol interference, only finite truncated approximation of the real 
pulse will be used.) Three cases, which are representative of practical 
situations, are considered. The results show that this new method has 
a significant improvement in accuracy over Chernoff bound. For exam- 
ple, we consider the 2400-baud DDD option of the Data Set 203 
operating over a channel having symmetrical delay distortion in excess 
of that of a worst-case C-2 line. A 5-tap mean-square equalizer is used 
by the receiver to equalize the channel. With a 14-dB input signal-to- 
noise ratio, the series expansion method provides a factor of 15 im- 
provement over the Chernoff bound in estimating the error probability 
at the equalizer output. 

The absolute convergence of the series expansion method is proved 
in Appendix A. An estimate of the terms required to reach the neigh- 
borhood of the true error probability is provided by equation (12). In 
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Fig. 7 — Error probability versus number of terms in equation (9). Fourth-order 
Chebyshev pulse, 11-pulse truncation approximation; sampling instant, I = 0.05 
r;(S/N) = 16 dB. 

o(0 = At COS («i Itl/r - <t>i) EXP (-01 Itl/T) 
+ A 2 COS (w, \t\/T - 2 ) EXP (- ai |t|/T). 

A, = 0.4023, A 2 = 0.7163, 

w, = 2.839, wo = 1-176, 

0! = 0.7553, 2 = 0.1602, 

a, = 0.4587, an = 1.107. 
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Fig. 8 — Error probabilities versus number of terms in equation (9). Fourth-order 
Chebvshev pulse; 11-pulse truncation approximation; sampling instant, t = 
0.2 f; (S/N) = 16 dB. 



actual systems, however, the true value is usually reached with only 
a small number of expansion terms. For example, in Fig. 3, the trunca- 
tion error is less than 2 X 10 -a after taking into account the 9th term 
of the series expansion (which involves the 18th moment of the inter- 
symbol interference) ; practically speaking, however, only three or 
four terms would be required for the series to converge in this example. 
In all the examples we considered, it is observed that a small error 
is assured by taking into account the first ten terms of the series. 

The convergence is somewhat slower if the ratio of intersymbol inter- 
ference to noise power [q(t )] is large (see Section IV, case 2.), as 
indicated in Figs. 5 and 8. Under this condition, either the intersymbol 
interference is so bad that the system is not of practical interest, or the 
input signal-to-noise ratio is so high that the Chernoff bound already 
assures that the system performance is acceptable. For both cases, 
there is no need to evaluate the error probability. 
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For computation purposes every system must be approximated by a 
finite-memory-system. Since the computations involved in this new 
method increase only linearly with the length of the memory, a good 
approximation of the true channel may be obtained without excessive 
computation. 
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Convergence of the Series Expansion Method 

In this Appendix, we shall prove that equation (9) is an absolutely 
convergent series. We know that 



Af a 



- f X 2k dF(x) 

•'nil X 

^ f (sup Xf k dF(x), 

•'all X 

= \z\r(to-m ir 



(20) 
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and 



H 2K+1 (x) = (-l) K 2 K+i [(2K - 1)!!]V2K + 1 

• exp (x 2 /2) ■ sin ( V±k + 3 x) + o(^Ag)] ' ( 21 )' 
Hence, 



P..K I ^ 



A ^Mv^-r i YV* i 






The ratio of So;r+2 to *S 2 jr is given by 



& 



V2K - 1 



fElK<o-«0 



S« (2X + 2) V27C + 1 



(22) 



(23) 



For K sufficiently large, equation (23) is always less than unity. There- 
fore equation (9) is an absolutely convergent series. 

APPENDIX B 

Derivation of the Recurrence Relations for the Moment of Intersymbol 
Interference 

It has been shown that the intersymbol interference converges 
absolutely to a random variable 8 X. The characteristic function of the 
random variable X is given by, 



*(„) = f e'* x dF(X), 

''all X 



-i + *af, + <|fjf, + ...+J|f*,+ .... (24) 



Therefore, we obtain 



* See Ref . 8. 
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*(0) = 1 



rf 2 <E(g>) 



d 2lc $(o>) 

dJ k 



= * 2 (0) = -M 2 , 



= *"(0) = (-l) k M 2k , 



(25) 



Since a t 's are identically and independently distributed random 
variables and with zero mean, 



M , = M s = • • • = M 2t+1 = 

and 



= for A; = 0, 1, 2, 



*(") = II coscor(t - IT), 



(26) 



(27) 



where a truncated iV-pulse-train approximation of the channel impulse 
response is assumed. 

The even-order moments could be obtained by differentiating equa- 
tion (27) 2k times, but the right hand side expressions could become 
untractable. However, if we differentiate equation (27) once and re- 
group the terms, we obtain the following, 



<S>'(co) = - 



= -[l>(<o - flOtajiwrft, - 0Oj-*toi 



- -/( u ) •*(«). 



(28) 



By successive differentiation of equation (28), a recurrence relation 
can now be obtained. Differentiating equation (28) 2k - 1 times, we 
obtain 



* afc (0)« -{t(o*_ l)*(0) 2a ""/ 2,_1 (0)} , 



where 



/ 2 '-'(0) -;TTp!l/to 

(/CO 



(29) 



(30) 



The power series expansion of tan wr(i — ^T 7 ) around origin is 
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tan «r(«o - IT) = orfe - IT) + M '° ~ ^ + • ■ ■ 

+ 2 " ( ^)7 1} i B >» i ("ft - /T » 2 " 1 + • • ■ > < 31 > 

where B 2k is the Bernoulli number. It can be seen that 



~ k tan ur(t - IT) 
aw 



= [rft, - mf - ( jjT +1) ; I #* +1 I, 



for /c = odd positive integers, (32a) 

= 0, 

for k — even positive integers. (32b) 

Thus, 



m -&iM 



cjk + l/Cfk+l i\ 

(F+ i) " ' B * +l l * Xi+1 



for A; = odd positive integers, (33a) 

= 0, 
for k = even positive integers. (33b) 



where 



Since 



I 



X* +] = Elr^-ffjr. (33c) 



M 2k = (-l)** a (0). (34) 

Combining equations (34) and (29), we obtain the recurrence relation 
for M 2k , 

M 2k = -{£ (^ Z j)(-l)Vl/ 2 a-.»/ 2, -'(0)} (35) 

where / 2,_1 (0) 's are given by equation (33a) . 

Knowing that M = 1, all the higher order moments can be obtained 
via equation (35) without the knowledge of dF{x). 
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